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Abstract 

This article represents a study aiming at developing: (1) an IBTEM which is capable to promote partnership 
between training providers and their client institutions, easy to understand, effective, efficient; and (2) an IBTEM 
implementation guide which is comprehensive, coherent, easy to understand, effective, and efficient. The method 
used in the study was Research and Development (R & D) suggested by D. Gall, P. Gall, and Borg (2007) that 
was simplified into three phases: pre-development, development, and validation phases. The pre-development 
phase was used to review related theories, interview resource persons, and collect relevant documents. The result 
of this phase was an IBTEM draft. The development phase was used to organize a focus group discussion (FGD) 
attended by participants from the Yogyakarta Education Quality Assurance Institute, Regional Offices of 
Education, school supervisors, principals, and teachers. The result of this phase was a revised IBTEM draft. The 
validation phase consisted of internal and external validations. The internal validation was done through a Delphi 
technique. The team in the Delphi involved experts in educational evaluation, educational management, and 
vocational education. The result of the internal validation phase was a theoretical IBTEM that was then validated 
externally. The external validation was conducted by carrying out limited field testing of the theoretical IBTEM 
involving three institutions: the Center for Development and Empowerment for Arts Teachers, Center for 
Development and Empowerment for Mathematics Teachers, and the Yogyakarta Education Quality Assurance 
Institute. The external validation phase resulted in a final IBTEM. The study results confirm that: (1) the 
developed IBTEM is capable to promote partnership between training providers and their client institutions, easy 
to understand, effective, and efficient; and (2) the developed IBTEM implementation guide which is 
comprehensive, coherent, easy to understand, effective, and efficient. 
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1. Introduction 

Becker (1993) states that education and training can raise the level of one’s earning and productivity that 
eventually improves her/his income over the cost of the training s/he might have spent. Noe (2010) is in line with 
Becker in that educational institutions nowadays have developed training models to innovatively improve the 
participants’ financial performance than those who have not participated in the trainings. However, both 
statements will come true provided that the trainings are properly designed as to meet the expected outcomes. In 
this perspective, impact-based training evaluation model (IBTEM) offers some promising innovations in 
promoting the training participants capacity to improve their level of contribution of institutional productivity. 

IBTEM is an evaluation model designed to measure the effectiveness of training in improving performance of 
the client’s institution. IBTEM is very keen on achieving the expected impact on client’s institution (rate of 
expectation-ROE). In IBTEM frame, evaluation is to be done in class by the training provider and at the 
workplace by the client’s institution under collaboration with or guidance from the training provider. 

The framework of IBTEM refers to the four levels of evaluation (Kirkpatrick, 1996). The first level (ELI) is 
REACTION, the second evaluation (EL2) is LEARNING, the third level (EL3) is BEHAVIOR, and the fourth 
(EL4) is RESULTS/IMPACTS. Reaction and learning evaluations are administered in class, while behavior and 
results are administered at the work place. Figure 1 below illustrates how the four levels of evaluation work. 
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Figure 1. Kirkpatrick’s four levels evaluation (1996) 


IBTEM works through the four levels of evaluation that cover both internal and external evaluation. The internal 
evaluation includes ELI and EL2, while the external includes EL3 and EL4. IBTEM design makes it possible to 
trace the alumni if they can apply their skills at the workplace and produce the expected outcomes as the results 
of training. This training model is believed to be able to improve employee skills, create new ideas, products, 
and promote excellent service to customers. The company needs to conduct effective trainings and staff career 
management continually for their prospective career development. The company also needs to motivate the 
employee and care for their needs so as to make them comfortable at the workplace (Noe, 2010). Therefore, this 
kind of training policy is not in a state of “should do” but, rather, “must do” for the company to survive and to be 
more competitive. 

D. Robinson and J. Robinson (1989) claim that most training providers are generally concerned with the training 
management in the classroom so that they administer ELI and EL2 only. This type of training is referred to as 
Training for Activity (TFA) in which case the training providers would never know if the participants have 
attained the intended skills and competence required for productivity impacts. Therefore, they not only need 
internal evaluation (ELI and EL2) but also external evaluation (EL3 and EL4) to measure the effectiveness of 
the training and impacts at the workplace. The latter is known as Training for Results or Impacts (TFI). 

In Indonesia, it refers to the Government Regulation no 79/2005 on development guidance and supervision of 
local government administration article 1, and item d points out that the central government is obliged to 
support local government in developing training program as a part of human resource development. For this 
purpose, all government institutions in Indonesia should include training program in annual plan. However, 
quality assurance and the effectiveness of the training program should remain a major issue. 

Some research findings show that training providers in Indonesia generally conduct evaluation on ELI and EL2 
only. Since they do not include EL3 and EL4, it is never clear if the expected training impacts have been 
achieved or not. The reasons for not doing EL3 and EL4 among others are: (1) low commitment of the training 
manager to the importance of the training; (2) limited resource persons in the field of training evaluation; (3) no 
budget sufficiently allocated for EL3 and EL4; (4) training managers do not care about the effectiveness and 
efficiency of the training (Tupamahu &Tjiptono, 2009; Steffenny & Praptiningsih, 2013). 

In response to those problems, we urge to develop and implement IBTEM to improve the quality of planning and 
implementation of training in Indonesia. This would include these facts: (1) the training content delivered in the 
classroom should be applicable at the workplace; (2) motivate the training participants and their institutions for 
IBTEM treats them as the subjects not object of training; (3) the participation of stakeholders in the training 
implementation should be intensified; (5) facilitate staff development to improve their skills in the area of training 
design and analysis of EL3 and EL4; (6) minimize or if possible eliminate overspending national budget and 
encounter the accusation that such training is a waste of fund; and (7) develop clean and professional government 
officials who are customer-oriented, capable of providing excellent services and promoting culture of sutainable 
quality improvement. 

Specifically, the purpose of this research and development is to generate: (1) an IBTEM which is capable to 
promote partnership between training providers and their client institutions, easy to understand, effective, 
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efficient; and (2) an IBTEM implementation guide which is comprehensive, coherent, easy to understand, 
effective, and efficient. 

2. Review of Related Literature 

Eseryel (2002) describe that tools and methodologies used in evaluation should help determine the 
effectiveness of instructional interventions, including the effectiveness of training program. However, in reality 
most training providers do not conduct the evaluation, especially on ET3 and ET4. Furthermore, McEvoy and 
Buller (1990) put forward possible explanations for inadequate evaluations that include: insufficient budget 
allocated; insufficient time allocated; lack of expertise; blind trust in training solutions; or lack of methods and 
tools. Also, partly the reason for training providers not to conduct ET3 and ET4 is the complexity of evaluation 
task. The scope of training evaluation is also wide, it may concern on learning transfer and organizational impact 
(as cited in Eseryel, 2002, p. 93). This literature review will describe four conceptual references: (1) training 
evaluation approach; (2) effective training; (3) training effectiveness; and (4) Kirkpatrick Business Partnership 
Model. 

2.1 Training Evaluation Approach 

According to Eseryel (2002), there are six categories of training evaluation approaches: (1) Goal-based evaluation, 
it is an evaluation approach that focuses primarily on the goal achievement; (2) Goal-free evaluation, it is an 
evaluation approach that does not only confirm to the achievement of training goal formulated, but also to know 
the added values (benefits) resulted from the training; (3) Responsive evaluation, it is an evaluation approach 
that focuses on aspects requested by a training partnership institution; (4) System evaluation, it is an evaluation 
approach that focuses on whether training or other intervention implemented is effective and efficient; (5) 
Professional review evaluation, it is an evaluation approach that is done by external evaluation experts; and (6) 
Quasi-legal approach, it is rarely done, and it is done based on legal basis ( actual court-of-inquiry format) to 
present evidence, testimony, and evaluate the training implementation and its product. 

Among the six approaches, the goal-based and system-based approaches are mostly used for training 
evaluation. The goal-based approach is used with reference to Kirkpatrik’s four level evaluation, and the 
system-based approach is used with reference to Context, Input, Process, Product (CIPP) model; Training 
Validation System (TVS); and Input, Process, Output, Outcomes (IPOO) model (Phillips, 1991). 

2.2 Effective Training 

The guideline book published by The United States Office of Personnel Management [US-OPM] (2011) explains 
that effective training takes place when the analysis of ELI and EL2 scores high and satisfactorily. This would 
mean that the response of the training participants is good and that they have acquired the expected skills or 
competence. Common indicators for an effective training would include programs offered, attendance, 
availability of study program catalog, reaction evaluation data, and the results of pretest and posttest. The former 
three indicators have low correlation with effective training, they just indicate the amount of time, effort, and 
money has been spent. The last two indicators are commonly used to confirm whether the training is effective. 

D. Robinson and J. Robinson (1989) identify at least five indicators for an effective training: (1) training 
providers are only concerned with internal evaluation (EL 1 and EL2), and the training outcomes are considered 
as the responsibility of the respective client’s institution; (2) the more training conducted, the better their 
performance; (3) the transfer of skills from class to workplace is next to nothing; (4) no training need analysis; 
and (5) there is no share training responsibility assigned to client institution. 

Effective training is important, but it is not enough since it just indicates an achievement in class. We never know 
if the alumni apply their knowledge and skills at the workplace and produce the expected outcomes. 

2.3 Training Effectiveness 

Training is effective when the acquired skills and knowledge are applicable at the workplace and foster the 
achievement of the institutional expectation and mission (US-OPM, 2011, p. 123). The mission achievement can 
be seen from the results of EL3 and EL4. The main goal of training is the accomplishment of the target 
performance of the client institution. The evaluation training approaches provide methods and tools to measure 
the accomplishment of the main goal. 

International Atomic Energy Agency [IAEA] (2003) describes the principles of effective training by stating that 
the facility is organized, staffed and managed to provide training that supports the mission achievement. Training 
requires a strong commitment from IAEA line management to support training programmes that contribute to 
fulfilling company goals and objectives. The training mission must be clear, and the individual roles and 
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responsibilities need to be defined (p. 2). 

Meanwhile D. Robinson and J. Robinson (1989) put forward five indicators of effective training that include (1) 
partnership with the client’s institution; (2) client institution needs; (3) assessment of the training alumni 
performance; (4) infrastructure, facility and environment that support the training program; (5) measuring the 
training results/impacts. 

New Zeland Quality Authority [NZQA] (2010) describes indicators for effective training in a wider perspective 
covering (1) ration of qualifications gained by a number of trainees employed in a company; (2) shorter time for 
trainers to attain qualifications; (3) survey of trainee satisfaction; (4) employer’s satisfaction and credits earned; 
(5) proportion of trainees proceeding to higher level of qualification within industry 

Presently many institutions have shifted their way of thinking from effective training to training effectiveness. 
This is triggered by their leaders’ question about the training contribution to the achievement of the program 
objectives and institution mission. 

2.4 Kirkpatrick Business Partnership Model 

J. Kirkpatrick and W. Kirkpatrick (2009) developed a Kirkpatrick Business Partnership Model [KBPM] which 
was a model of planning, implementation, and evaluation training. It is conducted on partnership basis with a 
client institution. There are five foundational principles as the basis of KBPM: (1) the end is the beginning; (2) 
return on expectation (ROE) is the ultimate indicatorof value; (3) business partnership is necessary to bring 
about positive ROE; (4) value must be created before it can be demonstrated; (5) a compelling chain of evidence 
demonstrates your bottom line value (as cited in US-OPM, 2011, p. 7). With a minor modification, schematically, 
KBPM was consists of 18 phases that begins with identification of client needs and the expectation of the 
training results followed by program planning and implementation, and then training evaluation, support and 
facilitate training alumni to apply training results at the work place, and ended by reporting analyzed results of 
training evaluation (ELI to EL4) to prove whether training results (rate of expectation [ROE]) has been achieved. 
In detail the 18 phases of KBPM is illustrated in the Figure 2. 
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Figure 2. Kirkpatrik Business Partnership Model (KBPM) 


3. Research Method 

The method used in this study is Research and Development by D. Gall, P. Gall, and Borg (2007) that simplify 
the process into three phases: Pre-development model. Development model, and Validation model. 

The pre-development phase is to review related theory, interview resource persons, and collect relevant 
documents. The main reference in this development is KBPM (2009) and Ministry of National Education of 
Indonesia regulation number 12, 2007 on school supervisor standard. The primary resource persons are from 
Yogyakarta Educational Quality Assurance Institute (YEQAI) as the responsible agency for organizing national 
training to strengthen school supervisor competence. The YEQAI resource persons are the Head of YEQAI, 
Head of the four sections, and two senior instructors. The result of this pre-development phase is an IBTEM 
draft. 

Model development phase is to conduct focus group discussion (FGD) involving 40 representatives of YEQAI as 
the training facilitators and five regions (Yogyakarta City, Bantul, Kulonprogo, Sleman, and Gunungkidul) 
District Office of Education (DOEs) as client institutions. YEQAI is represented by 10 members (the head, 4 
section heads, 5 senior instructors), while the respective DOEs is represented by four including one Program 
Planning Head, one supervisor coordinator, one supervisor, two training alumni supervisors, one school principal, 
and one teacher. The result of this development phase is a hypothetical IBTEM. 

Model validation phase is to conduct internal and external validations. Internal validation is done by using 
modified Delphi Technique that involves IBTEM experts. They are experts in Educational Evaluation, 
Educational Management, and Vocational Education. The result of this phase is an IBTEM that is ready to be 
validated externally. 
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External validation is done by conducting separate FGDs in three national training providers in Yogyakarta: 
Center of Development and Empowerment for Arts Teachers and Educational Personnel (CDEATEP), Center of 
Development and Empowerment for Mathematics Teachers and Educational Personnel (CDEMTEP); and YEQA. 
Ten Respondents in each institution include all section/division heads (four), senior instructors or functional staff 
representative (six). The selection technique for respondents in those institutions previously mentioned is 
purposive random sampling as an agreement between researchers and those institution heads above. The result of 
this external validation is the final IBTEM. 

4. Research Results 

This section describes the results of research and development that covers: (1) formulation of IBTEM draft; (2) 
development of IBTEM; and (3) validity test of IBTEM. 

4.1 Formation of IBTEM Draft 

As previously explained, the main reference in the development of IBTEM is KBPM (2009) that consists of 18 
phases. This KBPM was presented to YEQAI representatives in a meeting followed by discussion and question- 
answer sessions. Inputs from the meeting were accommodated, analyzed with reference to principles in 
goal-based training evaluation approach. The inputs from YEQAI can be described as the following. First, 
Phase 7 in Figure 3 (Deliver training program) used an approach of “in-on-in”, in which “first in” means deliver 
training in class, while “on” means implementation of the training material at the workplace, and “second in” 
means report and discuss the implementation of training material at the workplace. Second, to minimize the 
number of steps and make it easier to understand model, EL8 and EL9 are combined into one step, namely Step 
8. Third, with the same reason, EL11 and EL12 are also combined into one phase. Step 9. Graphically IBTEM 
draft is illustrated in Figure 3 below. 
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Figure 3. IBTEM draft 


154 

















www.ccsenet.org/ies 


International Education Studies 


Vol. 9, No. 2; 2016 


4.2 IBTEM Development 

IBTEM is developed by conducting FGD whose participants are those involved in national training for school 
supervisors capacity building. They are from YEQAI, Yogyakarta District Office of Education in Yogyakarta 
(YDOE ), school supervisors training alumni, school principals, and teachers. At this FGD, the research team 
describes the IBTEM draft and its characteristics, and then let the participants respond and give feedback for 
IBTEM development geared toward promoting partnership between training providers and their client 
institutions, easy to understand, effective and efficient. 

This FGD yielded the following response and feedback. The YEQAI participants critically review the national 
training plan. They explained that the national training is designed by the National Agency of Development and 
Empowerment for Teachers and Educational Personnel within the Ministry of Education and Culture in Jakarta. 
This training program is a national policy which is topdown in nature. The YEQAI serves as a training organizer 
at the provincial level in which case they do not have the authority to modify the training program including the 
evaluation approach to measure the effectiveness of training, 

Then, YEQAI explained that up to now YEQAI generally has made efforts to conduct effective trainings in line 
with the local authorities. Internal Evaluation (ELI and EL2) have been routinely conducted, but not evaluation at 
the workplace (EL3 and EL4). External Evaluations (at the workplace) have been articulated into training process 
in class by implementing in-on-in strategy in training class as Phase 7, which delivers training program. Report on 
training delivery process and analysis of evaluation results have always been written even though it have not yet in 
line with what is suggested in IBTEM draft. The things that have not yet been written refer to Phase 12 to know 
whether training results/impact have matched with the expected institution performances (ROE). 

Within the YDOE, the school supervisors, principals, and teachers claim that the training programs so far are 
organized by the central government. This top-down approach often makes the district people disoriented since 
the national policy and regulations frequently change from time to time. As a result, the training providers do not 
want to conduct need training identification (Phase 1), and agreement on expected training outcomes between 
training provider and client institution (Phase 2). They state later on that such IBTEM implementation would 
require a number of well-trained personnel in external evaluation (EL3 and EL4), time, budget, and commitment 
of stakeholders. 

YDOE and school agreed that IBTEM has to be comprehensive covering internal evaluation in training class 
(ELI and EL2) and external evaluation at the workplace (EL3 and EL4). Even so, IBTEM needs to be simple, 
easy to understand, easy to implement (practical), and of course, effective and efficient in measuring the 
achievement of training objective. In addition, IBTEM needs a guide book for training providers as well as for 
client institutions. 

Analysis of qulitative data collected by open questionnaire revealed that 67 percent of respondents respond 
positively to IBTEM. They agreed that as an evaluation model it promotes partnership between training provider 
and client institution, easy to understand, effective, and efficient. On the other hand, 33 percent of respondents 
stated that IBTEM is not practical and difficult to implement since it would need a lot of budget, human 
resources, and time. 

IBTEM also needs human resources who have specific knowledge and skills to conduct and analyze 
impact-based training evaluation, especially for EL3 and EL4. Furthermore, they also suggested that the number 
of implementation phases still need to be minimized. The implementation of IBTEM needs supports from many 
relevant parties, although field evaluation (EL3 and EL4) would be potentially subjective. 

All suggestions and criticisms for IBTEM development were accommodated, analyzed, to produce four inputs or 
feedbacks. First, to make simple model scheme without ignoring its essence, “Identify necessities for success and 
leading indicators” on top of KB PM scheme is eliminated because it should be inherent in phases of evaluation 
planning both in the training class and at the work place settings. Second, to avoid misunderstanding that IBTEM is 
an evaluation training model not planning training model, all phrases that are categorized in planning area should 
be eliminated. Consequently, Phase 6 in IBTEM Draft “Design and build training program and evaluation tools” is 
modified to “Design and build of training evaluation tools”. 

Third, Phase 7-“Deliver training program” is also eliminated, while in-on-in approach in model scheme is 
articulated in Phase 6 and in Phase 7. In the case of Phase l-“Identify business’ need”, it is considered a planning 
area, even so it is needed as a starting step in evaluation proses. The forum agreed that this Phase was left there and 
considered as Phase zero (0). Fourth, if possible, ROE is continued by calculating the rate of investment (ROI) of 
the training. These four inputs will be combined with other inputs from validity test of the mode that will be 
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discussed in the next section. All these inputs will be accommodated then analyzed, and the results will be 
illustrated in IBTEM Final (Figure 4). 

4.3 IBTEM Validation 

Model validation is done internally and externally. Internal validation is to examine whether conceptual 
framework and its content of IBTEM scientifically reflect an evaluation model. Therefore, IBTEM needs to be 
validated by relevant experts, while the purpose of external validation is to know whether IBTEM can be 
understood easily and practically implemented by participants at the workplace. 

4.3.1 Internal Validation 

Internal validation of IBTEM is done by administering a modified Delphi Technique that the IBTEM document 
is not sent to the expert team but the team is invited to a meeting to provide arguments for an agreement. Expert 
team consists of three respective experts in the area of Educational Evaluation, Educational management, and 
Vocational Education. The conceptual framework of IBTEM which reflects results/impact oriented for client 
institution, partnership, coherence, simple, easy to understand, effective, and efficient is presented to the team. At 
the end of the forum, the expert team agrees to provide the following recommendations. 

First, referring to a cycle model of quality management suggested by Deming (2000), the number of 
implementation phases in IBTEM needs to be classified into four categories: Planning Evaluation; Doing 
Evaluation; Checking (analyzing evaluation results and reporting); Action (revising or improving ROE). Second, 
when EL4 (Phase 8) does not meet institution expectation (ROE), Phase 8a needs to trace the causes. It is 
probably no facility which is needed to implement training results at the workplace or training alumni needs 
remedy in certain part of training material. Third, in line with the second recommendation, it is necessary to 
confirm whether results of EL4 (Phase 10) has met the agreed ROE (Phase 1). In this IBTEM scheme, this 
confirmation can be illustrated by an arrow. When training results (EL4) have met agreed ROE, client institution 
can raise its ROE and this would suggest that IBTEM is a cycle model of sustainable quality improvement in line 
with Kaizen principles (Akinyemi & Abiddin, 2013). 

These three recommendations resulted from internal validation will be combined with the four previous inputs 
from model development phase and produce the final IBTEM as shown in Figure 4 and ready to be externally 
validated. For the IBTEM implementation guide, the team agreed that the implementation guide should focus on 
four evaluation levels (ELI, EL2, EL3, and EL4), systematic description, analysis example for each level of 
evaluation, understandability, practicality, effectiveness, and efficiency. 
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Figure 4. Internal validation results of IBTEM 


4.3.2 External Validation 

The objective of external validation is to know whether IBTEM as an evaluation model can be generalized for a 
wider population. The external validation was done by conducting FGD involving three national training 
providers, namely The Center of Development and Empowerment Arts Teachers and Educational Personnel 
(CDEAT), Center of Development and Empowerment for Mathematics Teachers and Educational Personnel 
(CDEMT); and Yogyakarta Educational Quality Assurance Institution (YEQAI). Total number of respondent was 
30 people with 10 people from each institution. Respondents in each institution were represented by four (4) 
existing head divisions (general affairs, program planning, quality mapping, and facilitation) and six (6) 
function a I senior staff members. There were two things need to be validated: (1) IBTEM as an evaluation model; 
and (2) Implementation guide of IBTEM. 

4.3.2.1 External Validation of IBTEM as a Model 

Assessment of external validity to IBTEM as a model was done by distributing a closed and 
opened-questionnaire to respondents and asked them to assess on: (1) conceptual framework of IBTEM as an 
evaluation model depicted in Figure 4. The conceptual framework questionnaire comprises four aspects: 
partnership-oriented with client institution, easy to understand, effective, and efficient. 

To know respondent assessment to IBTEM, we used four Likert scales that range from strongly agree; agree; 
disagree; to strongly disagree weighing score respectively 4, 3, 2, and 1. To determine IBTEM feasibility as an 
evaluation model, we firstly calculated ideal values of mean and standard deviation based on weighing score 
used. Mean ideal (Xi) = !4 (1+4) = 2.5 and Deviation Standard ideal (SDi) = 1/6 (4-1) = 0.5. Feasibility criteria 
of IBTEM as an evaluation model refers to the normal distribution curve within criteria ranges as shown in Table 
1 below. 
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Table 1. Categorizing criteria for the feasibility of IBTEM as a model 


Criteria 

Interval Score 

Feasibility Category 

X>(X;+ 1.5 SDi) 

X> 3.25 

Very feasible 

X; < X < (X; +1.5 SDi) 

2.50 <X< 3.25 

Feasible 

(X,- 1.5 SDi) <X<Xi 

1.75 < X<2.50 

Not feasible 

X<(X ; - 1.5 SD|) 

X< 1.75 

Not very feasible 


Data collection tool used in this study was questionnaire containing a number of questions on main 
characteristics that should be part of IBTEM as an evaluation model. The main characteristics are 
partnership-oriented with client institution, easy to understand, effective and efficient. The Collected data and 
information were analyzed and the results were presented in Table 2. 


Table 2. Respondents assessment to IBTEM as an evaluation model 


IBTEM character aspects 

Likert Scale (1-4) 

Description 

Score 

Category 

IBTEM is capable to promote partnership with client institutions 

3.47 

Strongly Agree 

Very Feasible 

IBTEM is easy to understand 

3.17 

Agree 

Feasible 

IBTEM is effective to evaluate training impacts 

3.30 

StronglyAgree 

Very Feasible 

IBTEM is efficient to evaluate training impacts 

3.23 

Agree 

Feasible 

Overall Mean of IBTEM as a Model 

3.29 

StronglyAgree 

Very Feasible 


Note. 4 = Strongly Agree; 3 = Agree; 2 = Disagree; 1 = Strongly Disagree. 


Table 2 shows that mean score for all aspects (X) is 3.29 in the score range 1-4. It means that IBTEM is very 
feasible as an evaluation model. Mean score for each aspects of IBTEM is presented in Table 2 above. 

4.3.2.2 External Validity of IBTEM Implementation Guide 

Feasibility assessment to IBTEM implementation guide covers five aspects that are comprehensive, coherent, 
easy to understand, effective, and efficient. The results of data analysis on respondent assessment to IBTEM 
implementation guide are presented in Table 3 below. 

Table 3. Respondent assessment to IBTEM implementation guide 


LikertScale (1-4) 

IBTEM character aspects - Description 

Score Category 


The Guide is comprehensive to explains the implementation of 
ELI-EL4 

3.23 

Agree 

Comprehensive 

The Guide is coherent to explain the implementation of ELI-EL 4 

3.07 

Agree 

Coherent 

The Guide is easy to understand 

3.20 

Agree 

Easy to 
understand 

The Guide is effective to explain the implementation of EL1-EL4 

3.07 

Agree 

Effective 

The Guide is efficient to explain the implementation of EL1-EL4 

3.33 

Strongly 

Agree 

Very efficient 

Overall aspect Mean score of IBTEM 

3.20 

Agree 

Feasible 


Note. 4 = Strongly Agree; 3 = Agree; 2 = Disagree; 1 = Strongly Disagree. 
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Table 3 shows that the overall mean score is (X) = 3.20. It means that IBTEM implementation guide is overall 
feasible. 

5. Discussion 

Data analysis shows that IBTEM as a goal-based evaluation model is “ very> feasible This is [very] 
understandable because conceptually and technically IBTEM is really feasible with the following rationales. 
First, conceptually under IBTEM client institution is treated as subject, not as object, for that reason training is 
designed based on the spirit of partnership with goal/impact orientation as expected by client institution. 
Therefore, Phase 1 in IBTEM scheme (Figure 4) is to agree on defining outcome expected from the training. The 
same case for Phase 2 which is to agree on key behaviors and facilitation needed in implementing training results 
at the workplace. This means that client institution understands, agrees, and will be able to provide supports both 
policies and facilities needed by training alumni to implement what was learnt in training. 

Second, to make concept clear, locus of evaluation is categorized into two locations: internal evaluation that is 
done in class and external evaluation that is done at the workplace. These two categories are placed at top of 
IBTEM Final scheme (Figure 4). Internal evaluation covers participant responses to training implementation 
(Reaction-ELl) and participant skills/competence [ability] gained at the end of training (Learning-EL2). External 
evaluation covers the implementation of training material delivered at the workforce (Behavior-EL3) and 
results/impacts gained by client institution (Results-EL4). This concept emphasizes that training evaluation 
covers both internal and external evaluation; therefore, training is not completed yet when it covers only internal 
evaluation and will not affect the institution. External evaluation is inseparable from internal evaluation and this 
is something to be done. Without external evaluation we will not know the effectiveness of the training. 

Third, 'in-on-in'’ approach in training delivery. It is an articulation of training delivery in class and its behavior 
evaluation (EL3) at the workplace. As explained before, the first and second “in” are training delivery in class 
and “on” is an application of training material at the workplace, at the same time, is an evaluation of key 
behavior implementation (EL3) at the workplace. The intention of EL3 is to know whether training participants 
are able to implement training material at the work place. If not, questions need to be raised on some issues, such 
as training alumni are not confident, some facilities needed are not available. This “in-on-in” approach is also a 
solution of traditional training that focus only on class activities. This approach is employed with partnership 
spirit with client institution and it may become learning experience for client institution in conducting behavior 
evaluation (EL3). This third rationale is in line with the first that treats client institution as a subject not as an 
object, in which case, it is in the aspect of conducting behavior evaluation (EL3). 

Fourth, conceptually the 10 phases in IBTEM scheme (Figure 4) are categorized into three main phases: 
Planning, Doing; and checking. Planning consists of five phases (Phase 1-5). Doing consists of three phases 
(Phases 6-8), and Checking consists of two phases (Phases 9-10). This categorization is in line with Sinthukhot 
(2013). In his research model. Evaluation Model for Internal Quality Assurance, there are also three major 
phases: Preparation, Evaluation Procedure, and Reporting. However, these three major phases can still be added 
with Phase 4, that is, Action. This fourth phase makes up four major phases: Planning, Doing, Checking, and 
Action (PDCA), and forms a cycle of quality improvement management as suggested by Deming (2000). Action 
in this context means revising ROE when analysis on EL4 shows that training results in workplace is lower than 
ROE or increase ROE when training results at workplace has met or surpassed the target of training results. 

Implementation of IBTEM is a matter of enculturation, or treating client institution as subject, and making the 
training results meet/surpass client’s needs. In Total Quality Management language, it would be known as 
“customer satisfaction”. In the enculturation process, there are two out of 14 Deming’s principles that are 
relevant: “ Adopt the new philosophy ” and “ Put everybody in the institution to work accomplishing the 
transformation The new philosophy and transformation in IBTEM implementation is shifting paradigm from 
training for activity (TFA) to training for impact (TFI) as explained in the introduction section of this article. 

Fifth, technically IBTEM is easy to implement because it comprises only 10-phases compared to its main 
reference KBPM that comprises 18 phases. The reduction of phase number was done because IBTEM focus only 
on evaluation scheme so that phases that were related to planning domain were eliminated. The elimination of 
planning phases (e.g.. Phase 6-Design learning program and evaluation tools) was based on respondents’ 
feedback. The respondents indicated that they had difficulties to identify whether IBTEM draft (Figure 3) was 
training evaluation scheme or planning one. 

The phase reduction by combining several into one, e.g. in results evaluation report, was from the four phases 
(ELI, EL2, EL3, and EL4). The reduction of phases is to simplify IBTEM scheme. This is exactly what is 
suggested by Rogers (1995, p. 242). In his book Diffusion of Innovation, he said that the simpler the concept of 
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an innovation the easier to understand, accept, and practice (triability) and at the end concept will be more 
attractive to implement. 

The success of IBTEM implementation surely needs stakeholder’s understanding on training effectiveness and 
commitment of the leader and staff both in training provider and client institutions. For that reasons, the 
following topics will be discussed as references to the success IBTEM implementation and its future 
implications: (1) promoting partnership; (2) effective training and training effectiveness; (3) leader commitment; 
(4) an ideal training evaluation model; (5) IBTEM’s contribution to the development of science; and (6) 
challenges for IBTEM implementation. 

5 .1 Promoting Partnership 

Partnership in this case means collaboration among relevant parties in designing, administering, and evaluating 
training. The main parties in training partnership are training designer and client institution that sends training 
participants. Table 2 shows that mean score (X) for promoting partnership aspect is 3.47. It means that IBTEM 
as a training evaluation model is very partnership oriented. Phases 1 and 2 in Figure 3 conform to this 
description. Petersen (2011) defined “ partnership is an agreement when two or more people or groups work 
together toward mutual goals ... a partnership must benefit both sides for it to be truly effective” (p. 5). Good 
partnership will place both sides on the same level of position. Both sides become subjects in determining ROE. 
Contradictory in conventional training, designer (generally in public institutions) acts as subject and training 
participants are treated as object. This commonly happens in trainings conducted by central/national government 
following top-down polices. In partnership oriented training, client institution is involved and empowered 
especially in administering EL3 and EL4. 

5.2 Effective and Efficient Training 

According to Merriam-Webster Dictionary (1984) effective means having an expected or intended effect or 
producing a desired effect, while efficient means acting directly to produce an effect (pp. 418-419). Sharma 
(2009) explains the word effective means do the right things and efficient do the things right (p. 96). Table2 
shows mean score (X) for effective aspect is 3.30 that means IBTEM is “very effective” as an evaluation model. 
Mean score for efficiency is 3.23 that mean IBTEM is “efficient” to measure ROE. Therefore, IBTEM is 
functionally effective and efficient as an evaluation model for measuring achievement of ROE. To know 
training effectiveness we need to analyze EL4 whether this training result has met ROE. If it does not meet it, 
the cause(s) need to be traced through phase 8a in Figure 3 as explained in the earlier discussion. Especially in 
the context of private sector, training effectiveness should be followed up by calculating rate on investment 
(ROI). It is categorized as evaluation level 5 (EL5) as it was described by Phillips (2007) in Table 4 below. 


Table 4. Phillips’ five-level of evaluation 


Level 

Brief Description 

Reaction, Satisfaction & Planned Action 
-ELI 

Measures participant reaction to and satisfaction with the training 
program and participant’s plans for action. 

2. Learning of training material - EL2 

Measures skills and knowledge gains 

3. Application and Implementation of 
training material - EL3 

Measures changes on-the-job application, behavior change, and 
implementation. 

4. Business Impact - EL4 

Measures business impact 

5. Return on Investment (ROI) - EL5 

Compares the monetary value of the business outcomes with the 
costs of the training program 


Phillips (2007) explained that the ROI analysis procedure is done through four phases: Evaluation Planning, 
Data Collection, Data Analysis, and Reporting. Each phase is illustrated in following Figure. 
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Figure 5. ROI analysis model 


Mathematically ROI (in dollars) can be simply calculated using following formula. 

Total program benefits-Total program cost Net program benefits 

Total program cost Total program cost 

The FGD participants of public institutions could accept that they have not needed yet to conduct ROI or EL5, it 
would be enough to conduct EL4 to measure ROE achievement. This tendency is in line with the government 
policy financial accountability that focuses on the amount of money (in percentage) spent at end of the fiscal 
year. The higher the percentage of expenditure (ideally 100%) the better score they get on their institutional 
performance. When an institution cannot spend all annual budget assigned, the remaining funds/budget has to be 
handed over to Public Treasury Fund Office. The higher amount of funds cannot be spent by an institution, the 
worse its performance score. This performance assessment tradition drives public institutions to spend all annual 
budgets without considering its program efficiency. Among other programs, training is generally considered the 
easiest program to spend the allocated fund. It has been a common practice now that educational training is the 
easiest way to spend the money with little attention paid to EL3 and EL4 even ELI and EL2. Of cause, these 
institutions are not interested in training effectiveness and efficiency. 

5.3 Leader’s Commitment and Character 

There was consitency from one FGD to another in the way all participants agree that IBTEM implementation 
needs leader commitment than conventional training evaluation. Commitment in this case is especially on policy 
decision making followed by budget allocation, and time for the leader to be directly involved in IBTEM 
implementation. All these matters need to be articulated in vision, mission, and slogan of an organization. 
IBTEM should be considered as a new paradigm. Its implementation requires a paradigm shift as described 
earlier by Robinson and Robinson (1989) from conventional paradigm that focuses on activity in class or 
training for activity (TFA) to new paradigm which focus on institution impact or training for impact (TFI). 

Concerning leader role in paradigm shift, Podsakoff, MacKenzie, Moorman, and Fetter (1990) described that 
leadership research has shifted from one examining the effects of transactional leadership to the identification 
and examination of those behaviors exhibited by the leader that make followers more aware of the importance of 
values of task outcomes, active in their higher-order needs, and induce them to transcend self-interests, for the 
sake of the organization. These transformational behaviors are believed to augment the impact of leader 
behaviors on employee outcome variables, because followers feel trust and respect toward the leader and they 
are motivated to do more than they are expected to do (p. 108). 

This research description underpins that transformational leadership will accelerate the implementation of an 
innovation. Koehler and Pankowski (1997) explained further that transformational leadership is a process 
inspiring change and empowering followers to attain greater achievements, to improve themselves and to 
improve organization processes. It is an enabling process causing the followers to accept responsibility and the 
processes to which they are assigned.” Furthermore, Wahab, Fuad, Ismail, and Majid (2014) concluded that: 
“There was a significant relationship between the level of headmaster’s transformational leadership practiced 
and the teachers’job satisfaction and commitments (p. 40). Therefore, it is a need for all leaders in central, local, 
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and school levels in Indonesia to have commitment and characters of transformational leadership to support the 
implementation of IBTEM. 

Implementation of IBTEM as an innovation is a part of organizational development and it potentially splits the 
organization members into two groups: supporters and opponents. Goetch and Davis (1994) explained that 
supporters of innovation are usually able to see the benefit of the innovation both for organization and for its 
members. On the other side, opponents may not see those benefits or they may worry losing their roles, or 
positions or even reducing their income. To this group, it is suggested not to intimidate them but instead, 
approach them and explain rationally that IBTEM will increase organization performance and credibility to the 
eyes of stakeholders and finally bring benefit to employees. The supporters of IBTEM have to be able to 
convince the opponents so that this group shifts their beliefs or paradigms from TFA to TFI as previously 
explained. 

5 .4 Ideal Training Evaluation Model 

Kirkpatrick’s goal-based training evaluation model which was formulated in 1959, it consists of four levels has 
actually been developed in 1996 by Phillips (2007) into five levels as has been explained earlier. Then it was 
continually developed by Brinkerhoff (2006) into six levels of training evaluation. In facts, some scientists e.g. 
Becker (1993), O’Neill (2007), Hamblin (1974), and Bird et al (1970) with their CIRO (Context, Input, Reaction, 
Outcome) model have also developed this evaluation model. In details, the chronological development of this 
goal-based training evaluation model can be illustrated in the following Figure (https://www.google.co.id/image 
of brinkerhoff six level training evaluation). 


Learning Evaluation: 

From Kirkpatrick to Brinkerhoff 


Robert Brinkerhoff (2006) 
Gary Becker (1964. 1993) 


Mary Beth O'Neill (2007) 

Jack Phillips (1996) 

Anthony Hamblin (1974) 
CIRO (1970) 

Donald Kirkpatrick (1959) 


Level 6 

Wider Contribution 


To what extent has the 
participants' long-term 
potential been increased? 



What £-value have these 
results contributed to the 
organisation or individual? 


What results have been 
achieved and what can be 
attributed to the learning? 


What effect has this had on 
the participants' behaviour in 
the workplace? 


Level 1 

Reactions 


To what extent have skills, 
knowledge and attitudes 
changed? 


What was the participants' 
reaction to the intervention? 
Did it meet their expectations? 


C Facilitator.org.uk 2009 


Figure 6. Ideal training evaluation model 


This model is considered ideal since it evaluates long-term training impact on its alumni professional development 
(EL6), even though its implementation and results validation is not easy to do. There are many variables that 
influence professional development of the training alumni. This EL6 may be parallel with external effectiveness 
concept in education. Psacharopoulos and Loxley (1985) in Lockheed and Hanushek (1994) stated that “ External 
effectiveness has to do with the relationship between non-monetary inputs and monetary’ outputs. In education, 
this could refer to the degree to which certain pedagogical practices or school tracks affect student post-graduate 
salaries, other things equal. Studies contrasting the earnings of technical-vocational track graduate and the 
earnings of students graduating from academic tracks are examples ” (p. 6). The implementation of this evaluation 
model needs a tracer study in longitudinal approach. Consequently, the ET6 implementation needs knowledgeable 
and skillful people and more budget than IBTEM does. 

5.5 IBTEM Contribution to Science Development 

The contribution of this research is the development of IBTEM as an effective and efficient goal-based training 
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evaluation model that partnership oriented, effective, efficient, and applicable to Indonesian context. The 
conceptual model is simpler, easier to understand, and more practical to implement than its main reference 
KBPM. In the perspective of Total Quality Management, IBTEM consists of four main steps as a quality 
improvement cycle: Plan, Do, Check, Action (PDCA) as suggested by Deming (2000). 

In the context of organizational development, theoretically, the implementation of IBTEM should lead an 
organization to become learning organization, promote quality culture that eventually leads all employees to 
internalize values and principles of continuous quality improvement (continuous quality improvement) in line with 
Kaizen principle. 

5.6 The Challenges IBTEM Implementation 

Up to now, there is no much data and information available in Indonesia on the implementation of goal-based 
training evaluation. Tupamahu and Soetjito (2005), reported that generally companies in Indonesia have 
conducted training evaluation, but these evaluations were only conducted in class (ELI and EL2), and never 
reach evaluation at the workplace (EL3 and EL4). Steffenny and Praptiningsih (2013) recently did a research in 
private company PT Jaya Mas Mandiri Plus in Surabaya, and they reported that this company had conducted 
ELI to EL4, but no detail information reported except a little bit information that ELI and EL2 results were 
satisfactory, and those results were analyzed based on company point of view. The researcher suggested that the 
future training evaluation needs to be done and analyzed based on the data and information collected from 
training participant point of view. 

In short, not much information and data related to training evaluation available from both Indonesian private and 
state institutions. Similarly no reasons or causes identified for most institutions in Indonesia for not 
implementing goal-based training evaluation so far. 

In response to those factors, a training evaluation field guide published by US-OPM (2011) categorized those 
causes for not implementing goal-based training evaluation into six potential reasons and offered the following 
recommendations. First, since there is not enough budget available for outside class activities, review the 
existing training programs and substitute them with other training activities that produce the same results. For 
those training programs that cannot be substituted, find an alternative form of class training led by instructors 
with, for example, on-line system. This training form can be accompanied by e.g., technical assistance at 
workplace, job rotation, coaching, and teamwork. This saving in money and time from the alternative programs 
above can be allocated to conduct EL3 and EL4. 

Second, their leaders do not provide clear directions for doing EL3 and EL4. Therefore, information on 
impact-based training evaluation needs to be communicated openly to leaders that EL3 and EL4 will contribute 
to the achievement of organization mission. Third, leaders are only interested in conducting certain training but 
not in trying to develop other training models. It is recommended to share with leaders about ineffectiveness 
training and shows them research findings on effectiveness training using EL3 and EL4. Four, they do not have 
enough expertise except to conduct training in class. They should have done some things such as benchmarking, 
inviting training evaluation experts, or searching for relevant tutorial material in internet. 

Fifth, they do not have enough staff in conducting EL3 and EL4, but they can arrange and conduct ELI and EL2 
as efficient as possible to save resources. Develop ELI which is short and self-evaluation and use EL2 with 
retrospective pre- and post-assessments, not full pre and posttests. Allocate the saving budget for conducting EL3 
and EL4. Do external survey for EL3 -EL4, FGD only for specific mission and programs. Sixth, their training 
staffs are not confident to do training evaluation outside class. In this case, they could hold organization 
workgroups, and then invite an outside professional speaker(s) to speak on results/impacts oriented training 
evaluation and the benefit of training partnership. 

6. Conclusion 

With reference to the objectives, this research infers that IBTEM as a goal-based evaluation approach turned out 
to be partnership oriented between training provider and clients, easy to understand, effective, and efficient. 
Second, IBTEM implementation guide was really comprehensive, coherent, easy to understand, effective, and 
efficient as a field guide. The implication of this research are (1) the IBTEM implementation would need 
stakeholder’s commitment to shift their paradigm from training for activity (TFA) to training for impact (TFI); (2) 
there should be workshops or FGD for the selected stakeholders on how to implement IBTEM; and (3) Since 
IBTEM implementation needs human resources who have adequate knowledge and skills, therefore policies and 
budget should be proportionally allocated to conduct EL3 and EL4. 

Further research needs to be conducted to tryout IBTEM with a wider range of respondents and longitudinal in 
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nature for planning and implementation for both state and private institutions. The limitation of this research, 
among others, was on the selection of senior facilitators ( widyaiswara ) as respondents in YEQAI and in PPPPTK 
were not taken randomly, but through an objective negotiation involving the researcher team and the institution 
Heads. Research method used in this study was dominantly quantitative that rely on a questionnaire as collecting 
data technique. For the future, research on IBTEM should be balanced with qualitative approach or mixed. 
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